Bayesian Bandits, Secretaries, and Vanishing Computational Regret

We consider the finite-horizon multi-armed bandit problem under the standard stochastic assumption of independent priors over the reward distributions of the arms. We define a new notion of computational regret against the Bayesian optimum solution instead of worst-case against the true underlying distributions. We show that when the priors of the arms satisfy a log-concavity condition, there is a simple index-type policy that achieves per-step computational regret O ( log log T log T ) regardless of how the number n of arms relates to the time horizon T . This shows that the regret vanishes as T → ∞. As a corollary, we present an additive PTAS for this problem. This complements existing literature on worst-case regret bounds that only hold for the case when n is much smaller than T . The log-concavity condition is widely used and is satisfied by Beta, Gaussian, and Uniform priors, which are the most common in these settings. Our policy is far simpler to implement than the well-known Gittins index policy, which is also not optimal for the finite horizon case. We also give evidence that the log-concavity condition is necessary for the type of regret bounds we show. Finally, we show that our results also extend to the related “budgeted learning” and the secretary problems. ∗Departments of Management Science and Engineering and (by courtesy) Computer Science, Stanford University. Email: [email protected]. Research supported by an NSF ITR grant, the Stanford-KAUST alliance for academic excellence, and gifts from Google, Microsoft, and Cisco. †Department of Computer and Information Sciences, University of Pennsylvania, Philadelphia PA 19104-6389. Email: [email protected]. Research supported in part by an Alfred P. Sloan Research Fellowship, an NSF CAREER Award, and NSF Award CCF-0644119. ‡Department of Computer Science, Duke University, Durham NC 27708-0129. Research supported by an Alfred P. Sloan Research Fellowship, and by NSF via a CAREER award and grant CNS-0540347. Email: [email protected].